Thanks again for the opportunity to present at the Showcase.
The talk closed with a “Way forward” slide proposing VirtualiZarr + a forecast-specific GRIB parser (in the spirit of the HRRR-parser) to fix three problems in our GEFS / ECMWF pipeline:
- Zarr v2/v3: the pipeline sidestepped Zarr entirely.
- No Zarr codec integration: read Parquet with pandas, pulled chunks with obstore, and decoded with cfgrib via temp files: a manual, “Zarr-free” process with no native metadata, codecs or chunking.
- Performance — cfgrib cost ~120–470 ms per chunk because of temp-file I/O, where an in-memory Rust decoder should be far faster.
What we did instead: parquet references → Icechunk
Rather than write a VirtualiZarr parser, we converted our existing custom Parquet reference files based on kerchunk (the [url, offset, length] triplets pointing into the public GRIB archives) directly into Icechunk virtual stores, and registered gribberish as a Zarr v3 codec.
That combination turned out to close all three gaps on its own:
| Problem from the slide |
Status now |
| Zarr v2/v3, no native store |
 It is a Zarr v3 store — xr.open_zarr(..., zarr_format=3) |
| No codec integration |
gribberish is a registered Zarr v3 codec; GRIB2 decode happens transparently inside the chunk pipeline. Omit the import and you get UnknownCodecError: 'gribberish' |
| cfgrib temp-file cost |
In-memory Rust decode, ~1 s for a full 721×1440 field, no temp files, works with xarray/dask |
So ds["tp"].isel(...) now issues an S3 byte-range read and decodes through the codec natively. No pandas, no temp files.
A caveat, to be precise: Icechunk and VirtualiZarr are not really alternatives, they sit on different sides. Icechunk is the store layer, and that’s what delivered the wins above. VirtualiZarr is one way to build virtual references. Because our kerchunk-based reference builder already existed, the VirtualiZarr parser was simply not on our critical path. Its promised benefit, collapsing a large bespoke reference-builder into a small standard parser is a real one we have not captured, and remains attractive on the build side. Perhaps there is thinking to the grib-index-kerchunk method is more suitable to integrate with herbie in long term.
GEFS/IFS Icechunk virtual datasets (thanks to source.coop!)
Both stores are published on Source Cooperative for hosting the space.
https://source.coop/e4drr-project/forecasts
The chunks are virtual: only the store metadata lives on source.coop, while the data bytes stay in the public AWS buckets (noaa-gefs-pds, ecmwf-forecasts).
Readers need no credentials at all.
| product |
group(s) |
members × steps |
dates (00z) |
| NOAA GEFS |
0p25/00z |
30 × 81 (0–240 h / 3 h) |
2031 (2020-09-23 → 2026-04-15) |
| ECMWF IFS ENS |
0p4/00z, 49r1/00z, 50r1/00z |
51 × 85 |
401 / 794 / 51 |
ECMWF is split into three groups because the open-data schema changed eras
(0.4° 9-level → 0.25° 13-level → 0.25° 14-level dual-stream).
On “how big is it” — three different numbers
This one surprised us, so it’s worth stating plainly:
| measure |
GEFS |
ECMWF |
| store objects on source.coop |
5.4 GB |
15 GB |
| referenced GRIB (packed) |
~92 TB |
~620 TB |
dense float32 (ds.nbytes) |
697 TB |
2.79 PB |
ds.nbytes — the “Size: 697TB” xarray prints — is the dense, decoded, float32 size, assuming every cell exists uncompressed. It overstates the real data volume by roughly 4.5–7×. The honest number is the referenced GRIB: the bytes you’d actually pull if you read the whole store once.
Opening the store
Self-contained, no credentials. Swap PRODUCT to switch between the two stores:
# uv run --with 'icechunk>=2.1' --with 'zarr>=3.2' --with 'xarray>=2025.1' \
# --with 'gribberish>=1.4' --with s3fs --with numpy python
import icechunk
import numpy as np
import xarray as xr
import gribberish.zarr # registers the "gribberish" Zarr v3 codec
STORES = {
"gefs": dict(prefix="forecasts/noaa_gefs_aws_s3_icechunk_vd",
group="0p25/00z", container="s3://noaa-gefs-pds/"),
"ecmwf": dict(prefix="forecasts/ecmwf_ifs_ens_aws_s3_icechunk_vd",
group="50r1/00z", container="s3://ecmwf-forecasts/"),
}
PRODUCT = "gefs" # <-- swap to "ecmwf"
cfg = STORES[PRODUCT]
storage = icechunk.s3_storage(
bucket="e4drr-project", prefix=cfg["prefix"],
endpoint_url="https://data.source.coop", region="us-east-1",
anonymous=True, # public read of the store metadata
from_env=False, # ignore any stray AWS_* env vars
force_path_style=True, # source.coop needs path-style addressing
)
# authorize anonymous byte-range reads of the virtual chunks on AWS
auth = icechunk.containers_credentials(
{cfg["container"]: icechunk.s3_anonymous_credentials()})
# source.coop returns occasional transient 5xx; skip icechunk's eager
# manifest prefetch so a single bad GET doesn't fail the open
rc = icechunk.RepositoryConfig.default()
rc.manifest = icechunk.ManifestConfig(
preload=icechunk.ManifestPreloadConfig(max_total_refs=0, max_arrays_to_scan=0))
repo = icechunk.Repository.open(
storage, config=rc, authorize_virtual_chunk_access=auth)
ds = xr.open_zarr(repo.readonly_session("main").store, group=cfg["group"],
consolidated=False, zarr_format=3)
print(ds)
Some caveats
import gribberish.zarr is mandatory before reading anything — it registers
the codec. Without it even opening a group fails.- Arrays live under a group (
0p25/00z, 50r1/00z), never the root. Opening
the root returns an empty dataset.
- Turn off eager manifest preload on source.coop (as above). Icechunk otherwise
prefetches thousands of manifests in parallel, and source.coop’s occasional
transient 500s then fail the open. Reads themselves retry fine.
- Manifest memory scales with array chunk count. Reading any chunk loads that
array’s whole manifest (~200 bytes/ref). Our ECMWF 49r1/t has
794 dates × 51 members × 85 steps × 13 levels ≈ 44.7 M refs ≈ 9 GB, which
OOMs on a small machine. icechunk.ManifestSplittingConfig (splitting along
time at write time) is the fix
Code
Scripts, smoke tests and docs (anonymous open + a per-era coverage/verification
suite) are here:
grib-index-kerchunk/icechunk-dask at main · icpac-igad/grib-index-kerchunk · GitHub
Requesting community discussion and verification on this.
